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LONG-TERM  GOALS 

Our  long-term  goal  is  the  continuing  evolution  of  the  Regional  Oceanic  Modeling  System  (ROMS)  as 
a  multi-scale,  multi-process  model  and  its  utilization  for  studying  a  variety  of  oceanic  phenomena. 
The  dynamical  processes  span  a  range  from  turbulence  to  basin-scale  circulation.  A  complementary 
goal  is  to  explore,  document,  and  explain  the  nature  of  the  delicacies  of  the  simulations  for  highly 
turbulent  flows. 

OBJECTIVES 

Our  primary  objectives  are  code  improvements  and  oceanographic  simulation  studies  with  ROMS,  as 
well  as  with  Large  Eddy  Simulation  (LES)  for  boundary  layer  turbulence,  with  measurement 
comparisons  where  feasible.  The  targeted  phenomena  are  submesoscale  wakes,  fronts,  and  eddies; 
shelf  and  nearshore  currents;  internal  tides;  regional  and  Pacific  eddy-resolving  circulations  and  their 
low-frequency  variability;  mesoscale  ocean-atmosphere  coupling;  and  planetary  boundary  layers  with 
surface  gravity  waves.  A  parallel  objective  is  to  establish  the  characteristics  of  model  uncertainty  in 
ROMS  for  realistic  simulation  of  complex  flows,  as  an  intrinsic  model  contribution  to  analysis  and  to 
forecast  errors.  The  premise  is  that  defensible  alternative  model  designs  —  in  parameter  values, 
subgrid-scale  parameterizations,  resolution,  algorithms,  topography,  and  forcing  data  —  may  often 
provide  a  range  of  answers  comparable  to  the  model-measurement  discrepancies.  We  hypothesize 
that  some  part  of  the  model-to-measurement  and  model-to-model  differences  may  be  irreducibly 
inherent  in  the  mathematical  structure  of  modem  simulation  models. 

APPROACH 

To  address  these  problems  we  are  making  ROMS  more  of  a  multi-process,  multi-purpose,  multi-scale 
model  by  including  the  coupling  of  the  core  circulation  dynamics  to  surface  gravity  waves;  sediment 
resuspension  and  transport;  biogeochemistry  and  ecosystems;  non-hydrostatic  large-eddy  simulation; 
and  mesoscale  atmospheric  circulation,  and  by  providing  a  framework  for  data-assimilation  analyses 
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(led  by  others).  Our  major  algorithmic  objectives  are  cross-scale  grid-nesting  in  turbulent  flows; 
accurate  advection;  dynamically  adaptive  vertical  coordinates;  and  parameterization  of 
wave-breaking  and  other  mixing  effects.  Finally,  we  continue  to  further  improve  the  pre-  and 
post-processing  tools  and  on-line  documentation  for  ROMS.  To  test  the  hypothesis  of  irreducible 
uncertainty,  we  use  ROMS  to  evaluate  model  sensitivity  with  respect  to  plausible  variations  in  several 
test  configurations:  flow  past  idealized  sea  mounts;  realistic  Pacific  and  Atlantic  basin  circulations 
comparable  to  present  and  near- future  operational  analysis  configurations;  and  nested  regional 
subdomain  currents  and  eddies  with  mesoscale  and  submesoscale  flow-topography  interaction. 

WORK  COMPLETED 

In  the  past  year  we  have  worked  on  the  following  circulation  regimes  and  phenomena:  decadal 
Pacific  and  Atlantic  circulations;  equilibrium  regional  circulations  along  the  U.S.  West  Coast,  central 
Alaska,  Central  America,  South  America,  Solomon  Sea,  the  Kuroshio,  and  the  Gulf  Stream; 
mesoscale  eddy  buoyancy  fluxes;  submesoscale  surface  fronts,  filaments,  and  eddies;  topographic 
current  separation,  form  stress,  and  submesoscale  vortex  generation;  surface  waves  and  nearshore 
currents  and  internal  tides  in  Southern  California;  surface  wave  influences  on  the  turbulent  boundary 
layer;  bubbles  generated  by  wave  breaking;  and  mesoscale  air-sea  coupling.  The  algorithmic  work 
has  been  on  adapting  the  oceanic  equation  of  state  for  split-explicit  time  stepping  of  barotropic  and 
baroclinic  modes;  accurate  time-stepping  for  the  bottom  boundary  layer  in  shallow  water  (~  meters) 
and  wake  flows  past  topography;  open  boundary  conditions  for  highly  turbulent  flows;  incorporating 
surface  wave  effects  in  ROMS;  diagnosing  spurious  diapycnal  mixing  due  to  advection  errors  and 
designing  remedies;  a  new  model  of  a  size-distributed  bubble  population;  and  the  exploration  of 
several  test-bed  configurations  for  the  simulation  delicacy  investigation. 

RESULTS 

We  present  a  few  highlights.  The  publications  list  (papers  from  201 1  up  to  ones  likely  to  be  submitted 
in  2012)  provides  a  view  of  finalized  results. 

Basin  scale  circulation  and  eddy  transports:  Widespread  experience  in  basin-scale  oceanic 
modeling  indicates  a  high  degree  of  sensitivity  of  strong  currents  to  many  aspects  of  the  simulation 
configuration.  In  Lemarie  (2012a)  we  show  a  extensive  investigation  of  the  sensitivity  of  the 
circulation  in  the  Pacific  basin  to  model  formulation  and  forcing.  We  demonstrate  that  a  modified 
advection  algorithm  with  implicit  diffusion  rotated  to  align  with  isopycnal  surfaces  is  able  to  reduce 
the  amount  of  diapycnal  mixing  in  the  ocean  interior  results  in  a  great  improvement  in  the 
representation  of  tracer  distribution  of  water  masses  at  the  scales  of  the  large  scale  circulation.  The 
modified  advection  scheme,  KPP  boundary  layer  mixing,  and  a  new  vertical  grid  are  used  to 
implement  a  ROMS  basin-scale  0.5°  Pacific  configuration.  Comparisons  with  various  observational 
datasets  are  carried  out  to  evaluate  the  quality  of  our  results  in  terms  of  mixed-layer  depth,  tracer 
distribution,  horizontal  circulation,  and  seasonal  variability.  ROMS  gives  results  comparable  to 
existing  climate  models  in  this  way  showing  its  ability  to  deal  with  large-scale  oceanic  flows. 

Rotating  the  tracer  hyperdiffusion  from  an  iso- sigma  to  isopycnal  direction  proves  to  improve  the 
intermediate  water  representation  (Fig.  1)  and  the  Equatorial  Pacific  thermocline  structure  (Fig.  2). 
An  overall  tightening  of  the  thermocline  is  observed,  leading  to  a  more  realistic  Equatorial 
Undercurrent. 

We  have  repeated  this  process  for  the  Atlantic  basin.  Figure  3  shows  the  successfully  simulation 
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mean  path  of  the  Gulf  Stream  near  the  eastern  seaboard.  This  high  resolution  simulation  encompasses 
the  full  Atlantic  basin  with  its  southern  boundary  near  the  Antarctic  Circumpolar  Current  and  has  a 
mean  resolution  of  6  km.  In  addition  to  the  Gulf  Stream,  validation  shows  many  successfully 
simulation  aspects  of  the  Atlantic  circulation,  some  of  which  are  notoriously  difficult  to  model  such 
as  the  Gibraltar  overflow  waters  and  the  mid-ocean  Azores  Current. 

Buoyancy  and  tracer  fluxes  by  mesoscale  eddies  are  an  important  part  of  the  mean  circulation  and 
climate  in  many  places  in  the  ocean.  In  simulations  of  Eastern  Boundary  Systems  off  North  and 
South  America,  the  eddies  provide  an  offshore  cooling  and  onshore  warming  to  balance  the 
widespread  air-sea  heating  and  coastal  upwelling  cooling,  as  well  as  an  upward  re- stratifying 
buoyancy  flux.  This  occurs  through  an  eddy-induced  Lagrangian  mean  (bolus)  zonal  overturning 
circulation  counter  to  the  Eulerian  Ekman-upwelling  circulation  cell,  consistent  with  a 
parameterization  principle  as  release  of  the  available  potential  energy  of  the  alongshore  current 
(Colas  et  al.,  2012a,c).  In  addition,  eddy  nutrient  fluxes  along  offshore-descending  isopycnal  surfaces 
deplete  the  biological  productivity  of  the  upwelling  system  (Gruber  et  al.,  201 1). 

Surface  submesoscale  fronts  and  filaments:  Our  focus  is  on  situations  where  mesoscale  energy 
transfer  exchange  with  submesoscale  unbalanced  motions  is  likely  to  occur.  Due  to  an  rapidly 
expanding  set  of  investigation  on  oceanic  flows  at  submesoscales,  it  is  increasingly  clear  that  these 
phenomena  are  ubiquitous.  We  are  in  the  process  of  systematically  investigating  a  range  of 
submesoscale  surface  phenomena  that  fall  into  the  broad  categories  of  front,  filaments  and  vortical 
structures.  We  illustrate  with  a  few  examples. 

South  of  the  Gulf  Stream  separation  point  at  Cape  Hatteras,  interactions  between  the  North  wall  of 
the  Gulf  stream  and  undulation  of  the  shelf  break  lead  to  a  phenomena  known  as  a  shingle  eddy. 
Figure  4  shows  an  evolving  shingle  eddy  near  the  coast  of  North  Carolina.  Diagnostics  show  that 
these  structures  have  strong  interactions  with  the  shelf  break  leading  to  enhance  vertical  exchanges. 
Also  visible  in  Fig.  4  is  an  example  of  a  cold  filament  (inside  the  area  indicated  by  the  dashed 
rectangle).  This  filament,  which  has  strong  downward  velocities  at  its  core,  becomes  unstable  to 
instabilities  that  are  fueled  by  horizontal  shear  instead. 

After  separation,  the  North  Wall  of  the  Gulf  Stream,  exhibits  regions  with  intense,  small  scale 
vorticity  structures  (Fig.  5).  These  flows  are  driven  by  the  horizontal,  cyclonic  shear  of  the  Gulf 
Stream  and  again  are  characterized  by  large  vertical  exchange  velocities.  The  reason  for  asymmetry 
in  the  north  and  south  shear  regions  of  the  Gulf  Stream  is  the  fact  that  anti-cyclonic  shear  values  are 
strictly  limited  by  the  condition  for  inertial  instability.  This  limits  the  anti-cyclonic  values  to  lower 
values  than  —  /  while  no  such  limit  exists  for  the  cyclonic  shear  rates.  As  recently  as  the  winter  of 
2012,  these  structures  have  been  observed  within  the  context  of  the  ONR  LATMIX  program  (private 
communication,  Eric  D’Asaro). 

In  Fig.  6,  a  model  solution  of  a  mesoscale  eddy  in  the  Gulf  Stream  area  is  shown  in  normalized 
surface  relative  vorticity.  The  eddy  can  be  seen  to  be  unstable  to  much  smaller  scale  instabilities 
around  its  rim.  These  modes  are  hypothesized  to  be  mixed  layer  instabilities  that  are  essentially 
geostrophic  in  nature  but  have  a  smaller  horizontal  scale  due  to  the  reduced  scale  of  the  Rossby 
deformation  radius  in  the  mixed  layer.  The  small  scale  rim  instability  is  remarkably  persistent 
throughout  the  life  cycle  of  the  mesoscale  eddy.  Azimuthal  mode  decomposition  of  the  structure 
reveals  that  low  modes  that  are  energized  by  the  environmental  straining  field  act  to  re-energize  the 
eddy  available  potential  energy  field,  which  allows  the  rim  instability  that  is  driven  by  release  of 
potential  energy  to  persist.  This  is  an  example  of  the  capability  of  ROMS  to  compute  physically 
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relevant  solutions  that  can  be  related  to  small  scale,  complex  observed  phenomena. 

Interactions  with  topography:  In  recent  publications  we  have  shown  that  subsurface  currents  may 
interact  with  topographic  features  leading  to  the  formation  of  coherent  anti-cyclonic  subsurface 
mesoscale  eddies  and  locally  enhance  diapycnal  mixing  and  energy  dissipation  (Molemaker  et  al., 
2102).  More  recently,  we  have  focused  on  the  role  of  interactions  with  complex  topography  in  the 
global  budget  of  vorticity.  Specifically,  the  interaction  of  strong  western  boundary  currents  with 
topography  are  investigated.  Early  results  point  to  a  much  more  important  contribution  from  the 
torque  of  the  bottom  pressure  in  western  boundary  regions  than  previously  thought.  Figure  7  shows 
the  contribution  for  the  torque  of  bottom  pressure  on  the  barotropic  vorticity  equation  in  the  Gulf 
Stream  path.  For  illustration  purposes,  the  bottom  torque  is  only  shown  in  the  core  of  the  Gulf  Stream. 
While  not  uniform  in  sign,  the  integration  of  this  signal  along  the  path  of  the  Gulf  Stream  shows  a 
positive  contribution  of  barotropic  vorticity  that  is  an  order  of  magnitude  larger  than  the  contribution 
of  frictional  processes.  We  intend  to  replicate  this  diagnostic  in  several  other  areas  around  the  globe 
for  which  we  have  numerical  solutions  such  as  the  Solomon  Sea  and  the  Kuroshio  region  near  Japan. 
Surface  wave  effects  in  the  boundary  layer:  Surface  waves  influence  boundary  layer  currents 
through  vortex  force  and  tracer  advection  by  Stokes  drift  and  through  breakers  that  inject  momentum, 
energy,  and  bubbles.  In  simulations  of  the  canonical  Ekman  problem  (steady  wind,  equilibrium 
waves,  no  stratification),  we  find  that  waves  deepen  the  layer  by  enhanced  momentum  transport 
associated  with  Fangmuir  circulations,  and  the  most  apt  representation  is  as  a  mixing  of  the 
Lagrangian  mean  current  (McWilliams  et  al.,  2012).  In  the  transient  wind  and  wave  evolution  under  a 
hurricane,  wave-induced  variations  in  the  thermocline  entrainment  rate  enhance  surface  temperature 
differences  between  the  left  and  right  sides  of  the  storm  in  combination  with  the  effects  of 
inertial- wave  resonance  (Sullivan  et  al.,  2012).  Current  research  is  on  how  remotely  generated  swell 
waves  combine  with  local  wind  waves  to  modify  the  mean  Ekman  current.  Bubble  clouds  are  carried 
into  the  boundary  layer  in  the  down  welling  branches  of  Fangmuir  circulations;  their  penetration 
depth  increases  with  wind  speed,  and  the  model  simulations  match  observations  fairly  well  (Fig.  8). 
As  a  consequence  of  increasing  pressure  with  depth,  the  gas  dissolution  from  the  bubbles  leads  to 
supersaturation  even  in  air-sea  equilibrium.  Using  LES  simulations  at  different  wind  speeds  we 
developed  a  new  parameterization  of  air-sea  gas  exchange  and  find  that  this  supersaturation  effect 
causes  a  measurable  difference  even  in  abyssal  gas  concentration  (Fiang  et  al.,  2012c). 

Sensitivity  of  oceanic  simulations  to  forcing,  topography,  and  algorithms:  As  discussed  above  for 
the  Gulf  Stream,  we  find  great  sensitivity  of  the  path  of  the  Kuroshio  to  changes  in  topography  and 
land  boundaries,  both  local  and  remote  such  as  changes  to  the  land  mask  in  the  Solomon  sea. 
Additionally,  exploration  of  the  influences  of  the  wind  forcing  had  already  shown  great  sensitivity  of 
equatorial  current  structure  to  wind  forcing  (Lemarie  et  al.  ,  2012a).  Current  research  shows 
additional  sensitivity  of  the  path  of  the  Gulf  Stream  to  small  changes  in  remote  wind  forcing  patterns. 
Modifying  the  way  that  wind  forcing  is  extended  toward  land  boundaries  has  led  to  changes  in  the 
forcing  primarily  in  the  far  Northern  Atlantic  due  to  missing  observational  data  over  sea  ice.  These 
remote  changes  lead  to  changes  in  the  mean  path  of  the  Gulf  Stream  in  the  Cape  Hatteras  area.  In 
general,  it  is  clear  that  there  are  uncertainties  in  the  ingredients  of  an  ocean  circulation  model,  such  as 
the  different  surface  forcing  field,  that,  while  small,  lead  to  significant  changes  in  the  resulting 
circulation.  Finally,  the  Pacific  circulation  simulations  in  Lemarie  et  al.  (2012a)  contain 
demonstrations  of  algorithmic  sensitivities  for  the  advection,  KPP  parameterization,  and  vertical  grid 
(e.g.,  ,  Figs.  1-2).  We  are  reporting  on  these  forcing,  topographic,  and  algorithmic  sensitivities  within 
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the  context  of  the  dynamical  subjects  of  our  papers. 

Improving  efficiency  of  MPI  routines  and  computing  practices:  Recently  we  have  implemented 
several  measures  to  improve  code  performance,  robustness,  and  reproducibility,  and  to  adapt  to  the 
evolving  computing  environment,  outstanding  for  a  long  time  issue  of  the  running  ROMS  as  a 
multi-threaded  (Open  MP)  code  multi-core  node  vs.  the  performance  of  code  using  MPI  the  same 
problem  using  the  same  hardware  (i.e.,  MPI  running  only  a  single  node):  the  multi-threaded  code 
with  optimal  subdomain  partitioning  significantly  outperforms  MPI,  however  this  obviously  cannot 
vertically  integrated  momentum  equations  separately  using  a  much  smaller  time  step,  it  exchanges 
messages  of  two  significantly  different  sizes:  a  larger  one  associated  with  halo-exchange  for  3D 
arrays,  and  a  smaller  one  for  2D.  It  is  well  known  that  MPI  transmission  bandwidth  strongly  depends 
on  message  size  in  a  rather  complex  way,  Fig.  9.  Small  messages  are  severely  penalized  by  start-up 
latencies,  while  the  decline  for  very  large  messages  is  more  complicated  to  explain,  but  it  is  also  less 
important  for  our  purposes  because  message  sizes  exceeding  10, 000  would  occur  only  for  a 
hypothetically  large  subdomain  size  of  1003,  at  which  point  (actually  even  before  that,  as  follows 
from  the  past  experience  with  UCLA  ROMS)  it  is  advantageous  to  introduce  inner  partitioning  within 
each  MPI  subdomain  (this  is  motivated  by  cache  optimization),  which  will  also  split  such  very  large 
messages.  On  the  other  hand,  for  most  of  our  configurations  the  size  of  halo-exchange  messages  for 
2D  arrays  is  within  the  low  hundreds,  placing  them  to  the  lower-left  comer  on  Fig.  9,  far  below  the 
size  where  a  significant  fraction  of  theoretical  bandwidth  can  be  achieved.  To  counter  this  we  have 
combined  messages  exchanging  ghost  points  of  different  arrays  into  larger  messages  exchanging 
more  than  one  array  (up  to  4)  in  a  single  message  passing  session  whenever  it  is  possible.  This 
requires  logical  rearrangement  of  the  code;  however,  this  is  fortunately  coincident  with  the  structural 
requirements  associated  with  maintaining  both  MPI  and  Open  MP  parallelizations  within  the  same 
code  ( e.g all  message  passing  must  occur  only  at  the  very  end  before  Open  MP  synchronization 
events  (and  is  prohibited  elsewhere),  and  the  desirability  of  reducing  the  number  of  synchronizations 
for  efficiency,  achieved  by  admitting  a  small  amount  of  redundant  computations  near  the  subdomain 
edges  in  order  to  delay  synchronization  events).  After  this  revision  the  situations  where  halos  for  a 
single  2D  array  are  exchanged  in  separate  session  are  excluded  completely;  2D  arrays  exchanged 
within  the  3D  part  of  the  code  are  “attached”  to  the  3D  messages,  and  the  barotropic  mode  prognostic 
variables  (,u,v  are  exchanged  in  a  single  session.  In  practice  we  find  an  overall  performance  gain  of 
~  40%  for  a  1600x560x32  grid  running  on  32  nodes  (256  CPU  cores  total)  relative  to  the  same  code 
except  for  exchanging  arrays  individually. 

IMPACT/APPLICATIONS 

Geochemistry  and  Ecosystems:  An  important  community  use  for  ROMS  is  biogeochemisty: 
chemical  cycles,  water  quality,  blooms,  micro-nutrients,  larval  dispersal,  biome  transitions,  and 
coupling  to  higher  tropic  levels.  We  collaborate  with  Profs.  Keith  Stolzenbach  (UCLA),  Niki  Gruber 
(ETH),  Curtis  Deutsch  (UCLA),  David  Siegel  (UCSB),  and  Yusuke  Uchiyama  (Kobe). 

Data  Assimilation:  We  collaborate  with  Drs.  Zhinjin  Li  (JPL),  Yi  Chao  (Remote  Sensing  Solutions), 
and  Kayo  Ide  (U.  Maryland)  by  developing  model  configurations  for  targeted  regions  and  by 
consulting  on  the  data-assimilation  system  design  and  performance.  Current  quasi-operational,  3DVar 
applications  are  in  California  (SCCOOS  and  CenCOOS)  and  in  Alaska  (Prince  William  Sound). 
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TRANSITIONS 

ROMS  is  a  community  code  with  widespread  applications  (http://www.myroms.org). 

RELATED  PROJECTS 

Three  Integrated  Ocean  Observing  System  (IOOS)  regional  projects  for  California  and  Alaska 
(SCCOOS,  CenCOOS,  and  AOOS)  are  utilizing  ROMS  for  data  assimilation  analyses  and  forecasts. 
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FIGURES 


Salinity  [psu]  Salinity  [psu] 


Figure  1:  Temperature-salinity  relationships  for  four  distinct  water  masses:  ESPCW  (Eastern  South 
Pacific  Central  Water,  top, left),  WSPCW  ( Western  South  Pacific  Central  Water,  top, right),  SPEW 
(South  and  North  Pacific  Equatorial  Water,  bottom,  left),  and  WNPCW  (Western  North  Pacific 
Central  Water,  bottom,  right).  The  black  dotted  line  corresponds  to  the  SODA  climatology  used  to 
initialize  and  force  the  Pacific  model  at  the  boundaries.  The  ISO  (i.e.,  with  a  neutral  biharmonic  dif¬ 
fusion  operator)  and  SIG  (i.e.,  with  a  biharmonic  along  sigma  surfaces)  solutions  are  represented, 
respectively,  by  a  black  and  a  gray  line.  False  diapycnal  fluxes  are  much  reduced  in  ISO. 
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Figure  2:  Annual  climatology  of  equatorial  (2°S  -  2° N  averaged)  salinity  [PSU]  from  Johnson  etal. 
(2002)  (TAU  J2,  left),  and  ROMS  SIG  (i.e.  with  biharmonic  diffusion  along  sigma  surfaces,  middle) 
and  ISO  (i.e.  with  a  neutral  biharmonic,  right)  solutions.  Again,  ISO  is  more  accurate. 
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Figure  3:  Annual  mean  surface  current  speed  in  the  Northwest  Atlantic  showing  the  mean  path  of 
the  Gulf  Stream.  The  magenta  line  shows  the  0.3  m/s  contour  level  of  surface  geostrophic  velocities 
derived  from  the  AVISO  altimetry  data  set. 
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Figure  4:  Simulated  Sea  Surface  Temperature  (SST)  for  a  region  south  of  the  Gulf  Stream  sepa¬ 
ration  point  at  Cape  Hatteras.  Surface  velocities  are  the  white  vectors.  The  sea  surface  dynamic 
height  is  shown  with  thin  black  contour  lines.  A  mesoscale  “shingle”  eddy  is  present  on  the  shore¬ 
ward  side  of  the  Gulf  Stream,  its  birth  triggered  by  irregularities  on  the  shelf  break.  The  dashed 
black  rectangle  shows  a  submesoscale  cold  filament  intrusion  with  strong  downward  velocities  at  its 
center,  and  the  lower  panel  is  a  zoom.  This  filament  will  break  up  in  the  next  few  days  as  a  result  of 
horizontal  shear  instabilities. 


13 


longitude 


February  19  -  00:00 


longitude 


T(°C) 

A 


i 


19.0 

15.8 

12.6 

9.4 

6.2 


Figure  5:  Simulated  Sea  Surface  Temperature  (SST)  for  a  region  downstream  of  the  Gulf  Stream 
separation  point  at  Cape  Hatteras.  This  figures  shows  a  range  of  submesoscale  flow  that  exist  in 
the  presence  of  mesoscale  structures  such  as  the  warm-core  Gulf  Stream  Rings  to  the  north  and 
backward  streamers  off  of  meander  crests.  We  focus  on  the  submesoscale  instability  within  the 
rectangular  region,  with  a  zoom  in  the  lower  panel.  The  source  of  energy  for  this  instability  the 
horizontal  shear  in  the  North  Wall  of  the  Gulf  Stream.  It  has  enhanced  vertical  exchanges  and 
leads  to  relatively  strong  local  energy  dissipation. 


14 


50 

100 

150 

200 

250 

300 

350 

400 

450 

500 


100  200  300  400  500 


2 

1.5 

1 

0.5 

0 


-0.5 


-1 


-1.5 


-2 


Figure  6:  Surface  relative  vorticity  showing  an  anti-cyclonic  cold-core  Gulf  Stream  Ring  in  an  area 
southeast  of  Cape  Hatteras.  Clearly  visible  in  the  image  is  the  complex  submesoscale  structure  of 
vorticity  inside  the  ring,  which  arises  from  the  potential  energy  contained  in  the  radial  density  gradi¬ 
ents  of  the  Ring.  These  density  gradients  are  replenished  by  the  low  azimuthal  modes  of  oscillations 
within  the  Ring,  which,  in  turn,  are  energized  by  an  interaction  of  the  the  ambient  straining  field 
with  the  Ring. 
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Figure  7:  Time-mean  of  bottom  pressure  torque  [m/s2]  within  the  core  of  the  Gulf  Stream  ( green 
dashed  lines)  defined  by  the  peripheral  maximum  and  minimum  of  barotropiic  transport  (green 
contours  every  10  Sv).  While  of  both  signs  near  particular  topographic  features,  the  torque  is 
pervasively  positive  ( cyclonic )  along  the  Stream,  in  opposition  to  the  anticy clonic  wind  curl  in  the 
subtropical  gyre. 
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Figure  8:  Upper  panel:  An  instantaneous  snapshot  of  simulated  total  bubble  number  density  (num¬ 
ber  per  in’)  in  the  upper  ocean  at  ulO  =  17.5  m/s.  Lower  panel:  Relationship  between  bubble 
e-folding  depth  z0  and  wind  speed  ulO.  ( Black  dots:  observations  by  Vagle  et  al.  [J.G.R.,  2010]; 
Blue  circles:  bubbly  flow  solutions  assuming  waves  in  equilibrium  with  local  winds.)  ( Liang  et  al., 
2012c) 
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Figure  9:  Averaged  transmission  bandwidth  as  a  function  of  message  size.  Solid  red  curve:  using 
a  cluster  made  of  8  CPU-core  nodes  with  DDR  Infiniband  interconnect  between  them;  dashed 
blue:  2  CPUs  per  node,  Gigabit  network.  The  timings  were  obtained  on  two  different  clusters  by 
running  a  test  program  which  sends  and  receives  messages  of  a  fixed  size  in  a  pattern  similar  to 
the  actual  ROMS  halo-exchange  routines.  The  bandwidth  is  measured  by  tracking  the  accumulated 
size  of  all  messages  sent  by  a  single  MPI  process  and  dividing  by  the  total  wall-clock  time  ( hence  it  is 
bandwidth  per  MPI  node).  As  in  the  actual  ROMS  code,  each  MPI  process  receives  exactly  the  same 
amount  of  data  as  it  sends  during  this  test,  and  the  bandwidth  reported  here  is  only  for  sends  (or 
receives)  taken  separately.  The  test  involves  messages  sent-received  within  each  multi-CPU  node  in 
addition  to  the  messages  traveling  across  the  interconnecting  fabric.  Similarly  to  the  practical  usage 
of  ROMS,  we  explicitly  control  MPI  node  mapping  onto  hardware  nodes  to  maximize  the  amount  of 
data  sent  locally  within  each  hardware  node  and  minimize  messaging  across  the  network.  It  should 
be  noted  that  although  Infiniband  appears  on  this  plot  to  be  only  twice  or  slightly  more  better  than 
Gigabit,  it  should  be  kept  in  mind  that  there  are  8  processes  on  each  node  competing  for  the  access 
to  the  interface  vs.  2  processes  in  the  case  of  Gigabit.  The  actual  difference  in  bandwidth  aggregate 
per  network  interface  is  more  dramatic. 
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